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Preface 


This  report  contains  the  result  of  an  investigation 
into  the  ability  of  a time  domain  pattern  recognition  algo- 
rithm to  classify  targets  by  analysis  of  their  radar  signa- 
tures. The  time  domain  procedure  evaluated  is  the  Frequency 
of  Occurrence  of  Binary  Words  method,  which  was  developed  by 
Peter  W.  Becker  while  engaged  with  the  General  Electric 
Company  in  Syracuse,  New  York.  The  recognition  problem  was 
jointly  proposed  by  Dr.  Matthew  Kabrisky,  Professor  of 
Electrical  Engineering  at  the  Air  Force  Institute  of 
Technology  (AFIT) , and  Major  C.  V.  Stewart,  an  AFIT  doctoral 
student  who  is  preparing  a dissertation  in  the  same  area. 

I wish  to  express  my  sincere  appreciation  to  Major 
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Stewart  for  providing  the  data  and  for  his  tutelage  through- 
out the  problem.  I also  wish  to  thank  Major  R.  A.  Gagnon 
of  the  Air  Force  Avionics  Laboratory  for  his  many  timely 
suggestions.  Finally,  I am  particularly  indebted  to 
Dr.  Kabrisky,  who  served  as  my  thesis  advisor,  for  his 
suggestions,  encouragements,  and  infinite  patience  during 
the  course  of  this  investigation. 

Delbert  G.  Kulchak 
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The  purpose  of  this  investigation  is  to  examine  a time 
domain  pattern  recognition  algorithm  which  will  identify 
targets  by  analysis  of  their  radar  signatures.  The  essence 
of  the  algorithm  is  a conversion  of  an  analog  waveform  to  a 
binary  sequence,  from  which  binary  words  are  selected  as 
attributes.  The  selection  of  attributes  is  a heuristic, 
nonexhaustive  process,  and  each  target  is  eventually  de- 
scribed in  a statistical  sense  by  the  frequency  of  occur- 
rence of  the  binary  word  attributes.  A relationship  between 
this  Frequency  of  Occurrence  of  Binary  Words  method  and 
the  Fourier  transform  is  also  shown. 
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TARGET  CLASSIFICATION  BY  TIME  DOMAIN 


ANALYSIS  OF  RADAR  SIGNATURES 

I . Introduction 

Recent  investigations  of  the  radar  returns  from  illumi- 
nated targets  indicate  that  information  may  be  present  in 
the  signatures  which  could  serve  to  distinguish  between  dif- 
ferent categories  of  targets.  One  is  often  tempted  to  pur- 
sue the  analysis  of  radar  signatures  via  frequency  domain 
techniques;  the  Fourier  transform  is  an  extremely  powerful 
signal  processing  tool  and  has  been  used  successfully  in 
many  radar  signal  processing  applications.  An  alternate 
approach,  however,  would  be  to  examine  the  signatures  entire- 
ly in  the  time  domain.  Time  domain  techniques  tend  to  be 
somewhat  easier  to  implement  in  programming  languages  and 
also  tend  to  be  more  suitable  for  minicomputer  and  micro- 
processor applications.  If  a suitable  time  domain  pattern 
recognition  algorithm  could  be  found,  it  might  represent  a 
considerable  advantage  in  these  respects  when  compared  to 
frequency  domain  solutions  of  the  same  problem. 

Radar  returns  of  three  targets  were  recorded  under  a 
variety  of  conditions  associated  with  target  velocity  and 
antenna  orientation.  The  targets  were  illuminated  at  16 
« different  aspect  angles,  0 to  360  degrees,  with  each  success- 
slve  observation  spaced  22.5  degrees  from  the  previous  ob- 
servation. With  few  exceptions,  target  velocities  were 


1 


■ t 

relatively  constant  during  the  observations.  The  returns 
were  digitized  and  recorded  on  magnetic  tape,  and  used  in 
this  form  as  the  raw  data  for  the  pattern  recognition  algo- 
rithm. 

The  time  domain  procedure  evaluated  in  this  report  it. 
the  Frequency  of  Occurrence  of  Binary  Words  (FOBW)  (Ref  1) 
method.  The  FOBW  method  is  quite  simple  to  Implement  and 
requires  only  that  the  data  under  consideration  be  capable 
of  being  represented  as  a binary  sequence.  This  sequence  is 
then  scanned  for  the  frequency  of  occurrence  of  specific 
binary  words,  with  the  expectation  that  members  of  a given 
class  will  exhibit  different  frequencies  of  occurrence  for 
some  words  than  will  members  of  another  class. 


II 


Pattern  Recognition 


Pattern  recognition  is  becoming  an  increasingly  valuable 
tool  in  the  mechanization  of  tasks  which  previously  have  been 
performed  only  by  humans.  The  applications  of  machines  with 
the  ability  to  classify  and  sort  data  are  almost  too  numerous 
to  mention,  and  include  such  diverse  subjects  as  speech  recog- 
nition and  synthesis,  waveform  interpretation,  character 
recognition,  and  photographic  interpretation.  Contributions 
to  the  growth  of  pattern  recognition  have  come  from  many 
equally  diverse  disciplines,  and,  in  the  eyes  of  many  experts, 
the  ultimate  goal  of  emulating  certain  classes  of  human  re- 
sponse to  sensory  inputs  may  be  realized  in  the  not-too- 
distant  future. 

A typical  pattern  recognition  system  might  resemble  the 
block  diagram  shown  in  Fig.  1.  The  analog  real-world  energy 
of  interest  is  first  detected  by  an  appropriate  sensor  and 
transmitted  to  a signal  preprocessor.  In  the  preprocessor, 
the  detected  energy  is  mapped  into  a finite  dimensional 
pattern  space.  From  the  information  available  in  the  pattern 
space,  certain  features  are  extracted  which  the  designer  sus- 
pects are  representative  of  the  classes  of  patterns  he  wishes 
to  separate.  At  this  point,  the  designer  has  significantly 
reduced  the  dimensionality  of  the  space  which  contains  the 
' energy  of  interest.  The  selected  features  are  then  presented 
to  a classifier,  which  uses  some  decision  rule  to  separate 
the  features  into  two  or  more  classes.  It  is  generally 
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assumed  that  the  classifier  operates  in  a deterministic 
fashion;  there  appear  to  be  few  practical  applications  for 


classifiers  with  random  outputs. 

In  the  design  of  an  actual  pattern  recognition  system, 
each  element  of  the  typical  system  shown  in  Fig.  1 is  usu- 
ally selected  in  accordance  with  the  specific  problem  at 

hand.  The  sensor  is  selected  by  consideration  of  the  physics 

'■e 

associated  with  the  observed  energy,  the  preprocessing  should 
enhance  the  data  to  be  separated,  and,  most  importantly,  the 
feature  extractor  must  be  capable  of  determining  what  best 
represents  the  various  classes.  In  addition  to  the  selection 
of  pattern  recognition  systems  components,  one  must  also 
generate  a representative  data  base,  select  training  and 
test  sets,  evaluate  the  performance  of  the  system,  and  make 
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as  many  iterations  in  the  entire  process  as  is  required  for 
satisfactory  operation. 

As  mentioned  above,  the  feature  selection  phase  of  the 
pattern  recognition  process  is  nearly  always  the  most  cru- 
cial step.  Unfortunately,  it  is  often  the  most  difficult, 
also.  Poorly  selected  features  generally  add  to  the  complex- 
ity of  the  classifier  and  decision  rule  and  may  make  success- 
ful recognition  impossible.  There  is  very  little  theory, 
however,  to  guide  one  in  a decision  as  to  what  constitutes 
effective  features  in  a given  problem.  The  search  for  effec- 
tive attributes  often  is  a heuristic  affair,  and  the  designer 
usually  relies  on  his  own  intuition  after  considering  th' 
physical  nature  of  the  classification  problem. 

The  separation  of  the  data  base  into  training  and  test 
sets  can  also  be  a nontrivial  task.  The  training  data  must 
be  sufficiently  varied  and  inclusive  to  allow  the  recognition 
system  to  practice,  yet  the  test  data  must  be  sufficiently 
different  to  give  credence  to  the  final  results.  It  would 
serve  little  purpose,  for  example,  to  draw  the  test  data  from 
the  design  data  since  the  recognition  system  could  then  have 
an  artificially  high  success  rate. 

A performance  evaluation  is  the  final  step  in  the  recog- 
nition process.  Some  method  must  be  chosen  which  would  indi- 
cate whether  or  not  the  performance  of  the  system  is  accept- 
able. The  indices  of  performance  are  often  specified  before- 
hand, and,  after  a fashion,  influence  the  choice  of  a decision 
rule.  If  one  is  concerned  only  with  minimizing  the  overall 
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risk,  for  example,  the  decision  rule  may  be  an  extension 
of  Bayes'  criterion.  If  minimal  false  alarm  or  reject  rates 
are  also  of  primary  concern,  however,  the  Neyman  Pearson 
method  or  one's  own  criterion  may  be  more  appropriate. 
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III.  The  Frequency  of  Occurrence  of 
Binary  Words  Method 


Introduction 

The  Frequency  of  Occurrence  of  Binary  Words  method  is 
elegantly  simple  to  implement  and,  with  few  constraints, 
requires  only  that  the  data  under  analysis  be  capable  of 
being  represented  as  a binary  sequence.  The  binary  sequence 
is  scanned  for  the  frequency  of  occurrence  of  specific 
binary  words,  and  these  frequencies  are  then  used  as  attri- 
butes in  the  pattern  recognizer.  It  is  assumed  in  the  re- 
mainder of  this  discussion  that  the  raw  data  consists  of 
analog  or  discrete  analog  waveforms,  although  this  need  not 
be  the  case  in  practice. 

When  a waveform  has  been  coded  in  an  appropriate  fashion, 
it  can  be  represented  as  a binary  sequence.  A typical  coding 
algorithm  might  consist  of  sampling  the  waveform  at  a con- 
stant rate  and  coding  all  positive  samples  as  a 1 and  all 
nonpositive  samples  as  a 0.  The  resulting  binary  sequence 
may  be  regarded  as  a string  of  adjacent  bits,  where  each  pair 
of  adjacent  bits  is  called  a digram  (Ref  1:76).  In  a simi- 
lar fashion,  trigrams,  tetragrams,  pentagrams,  and  n-grams 
can  be  formed  by  sets  of  3,  4,  5,  ...,  n adjacent  bits. 

Each  n-gram  can  occur  as  any  or  all  of  its  2n  permuta- 
tions in  a given  binary  sequence.  The  sequence  11111000000, 
for  example,  contains  11  bits  and  10  pairs  of  adjacent  bits. 

If  n is  equal  to  2,  four  digrams  occur,  or  have  a frequency 
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of  occurrence,  as  shown  below 
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In  the  notation  above,  f represents  the  frequency  of  occur- 
rence and  the  subscripts  refer  to  the  two  adjacent  bits  In 
each  digram  permutation.  One  may  note  that  the  sum  of  the 
frequencies  of  occurrence  for  any  given  n-gram  is  unity. 

This  may  be  seen  more  readily  in  Fig.  2,  which  shows  the 
digram,  trigram,  and  tetragram  frequencies  of  occurrence 
in  a sample  32  bit  sequence. 

Digrams  which  contain  delays  also  may  be  present  in  a 
given  binary  sequence.  A delayed  digram  (Ref  1:79)  is  simply 
a digram  in  which  the  two  bits  making  up  the  digram  are  sep- 
arated by  a variable  number  of  "don't  care"  bits,  e.g.,  the 
delayed  digram  l-m-0  consists  of  a 1 followed  by  m sampling 
intervals  (or  m minus  1 bits),  which  in  turn  is  followed  by 
a 0.  The  delayed  digram  1-3-0,  for  example,  could  take  any 
of  the  following  forms:  1000,  1010,  1100,  or  1110,  and 
occurs  once  in  the  five  bit  sequence  10101.  The  delayed 
digram  l-m-0  might  also  be  considered  to  be  an  (m-fl)-gram 
with  don't  care  states  for  all  but  the  first  and  last  bits. 
Figure  3 depicts  some  delayed  digram  frequencies  of  occur- 
rence for  the  same  binary  sequence  shown  in  Fig.  2. 
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16  Tetragrams 
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29 
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31 


Fig.  2.  n-gram  Frequencies  of  Occurrence  for  the 
32  Bit  Sequence:  11111100011111000000000000000011 
(Adapted  from  Ref  1). 

The  frequencies  of  occurrence  of  the  n-grams  and  delayed 
digrams  constitute  the  features  which  are  used  to  separate 
the  various  classes.  If  the  Initial  problem  is  amenable  to 
solution  by  the  F0BW  method,  one  may  expect  that  the  frequen- 
cies of  occurrence  (referred  to  simply  as  frequencies  here- 
after) of  particular  n-grams  and  delayed  digrams  will  remain 
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Fig.  3.  Delayed  Digram  Frequencies  of  Occurrence  for 
the  32  Bit  Sequence:  11111100011111000000000000000011 
(Adapted  from  Ref  1). 


essentially  constant  among  members  of  the  same  class  and  yet 
will  be  sufficiently  different  from  members  of  another  class 
to  allow  a decision  rule  to  be  made. 

Sequence  Length 

One  of  the  constraints  imposed  by  the  FOBW  method  con- 
cerns the  physical  length  of  the  coded  binary  sequence;  the 
sequence  should  be  long  enough  to  permit  the  computation  of 
binary  word  frequencies  to  a desired  accuracy.  If  the  origi- 
• nal  waveform  is  generated  by  an  ergodic  process  and  sampled 
at  a harmonic  of  its  period,  a single  period  in  the  binary 
sequence  will  provide  as  accurate  a calculation  as  is 
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possible.  If  the  original  waveform  Is  generated  by  a sto- 
chastic process,  however,  the  actual  length  of  the  coded 
binary  sequence  becomes  more  significant.  In  such  a case, 
a determination  of  what  constitutes  adequate  length  may  be 
made  in  the  following  fashion  (Ref  1:  91-94). 

If  a binary  sequence  containing  L bits  is  scanned  for 
an  n-gram  frequency  of  occurrence,  and  if  the  intervals 
between  observations  are  sufficiently  long,  the  n-gram 
frequencies  may  be  assumed  to  be  uncorrelated.  If  the 
n-gram  of  interest  is  of  the  form  \p,  the  scanning  process 
will  reveal  T total  n-grams,  of  which  y are  of  the  desired 
form  ip.  The  probability  that  any  observed  n-gram  is  of 
the  form  ip  may  be  estimated  by  y/T.  Davenport  and  Root 
(Ref  4:  84-85)  demonstrate  that  y/T  has  the  expected  value 


f 


and  the  variance 


E[y/r]  - 


(1) 


Varty/T] 


y*-y 


(2) 


It  is  assumed  that  P^  does  not  change  with  time,  and  one 
may  note  that  y/F  has  a Bernoulli  distribution  regardless 
of  the  waveform  process.  Substituting  these  expressions 
into  the  Tchebycheff  inequality,  one  obtains 


(3) 
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As  the  sequence  length  L Increases  without  limit,  the  total 
number  of  observed  n-grams  also  increases  without  limit. 

The  variance  in  Eq  (2)  can  be  seen  to  approach  zero  as  L 
becomes  sufficiently  large,  and,  by  the  Bernoulli  theorem 
(Ref  4:85),  the  value  of  y/T  converges  in  probability  to  P^, 
also . 

The  inequality  in  Eq  (3)  can  be  used  to  determine  if 
the  estimates  of  associated  with  the  particular  length 
of  the  binary  sequence  are  within  a desired  confidence  in- 
terval. If  the  estimates  are  not  acceptable,  the  binary 

. 

sequence  length  may  be  increased  until  the  estimates  fall 
within  the  desired  interval. 

To  this  point,  it  has  been  assumed  that  the  waveforms 
are  time  stationary.  The  binary  word  frequencies  of  occur- 
rence may  still  be  effective  attributes,  however,  even  if 
this  requirement  is  not  met.  If  the  parameter  values  for 
the  process  which  generated  the  original  waveform  change 
with  time  in  a well  defined  fashion,  the  instantaneous 
binary  word  frequencies  will  also  vary  in  the  same  manner. 
Each  observed  frequency  of  occurrence  would  then  constitute 
a time  average  that  may  in  itself  be  an  effective  attribute 
(Ref  1:94). 

The  FOBW  Method  and  the  Fourier  Transform 

A particularly  interesting  feature  of  the  FOBW  method 
is  its  relationship  to  the  Fourier  transform.  One  might 
suspect  intuitively  that  such  a relationship  does  exist  by 
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considering  the  information  present  in  symmetrical  and  un- 


symmetrical  binary  words.  If  a given  binary  sequence  is 

scanned  first  from  left  to  right  and  then  from  right  to  left, 

two  sets  of  frequencies  will  be  recorded.  It  can  be  verified 

that  the  frequency  of  a symmetrical  word  or  digram,  e.g., 

f,-.,  or  f-  , will  be  the  same  regardless  of  the 

1001  O-m-l-m-0 

direction  taken  during  the  scanning  process.  In  general, 

however,  the  frequencies  of  unsymmetr ical  words  or  digrams, 

e.g.,  f11ft1  or  fn  . . , will  not  be  the  same. 

° 1101  0-m- 1-m- 1 

One  interpretation  of  these  results  (Ref  1:109)  suggests 
that  the  frequencies  of  symmetrical  words  contain  information 
similar  to  that  obtained  from  power  spectra,  autocorrelation, 
and  other  amplitude  related  functions  which  are  independent 
of  the  direction  in  which  time  is  measured.  The  frequencies 
of  unsymmetr ical  words,  on  the  other  hand,  may  be  construed 
to  represent  phase  information,  which  is  sensitive  to  the 
direction  in  which  time  is  measured.  One  may  observe  that 
amplitude  and  phase  are  the  two  terms  which  define  the 
Fourier  transform  F(f)  when  it  is  written  in  the  form 

F(f)  - A(f ) exp (j  0 (f ) ) (4) 

where  A(f)  is  the  Fourier  amplitude  of  F(f),  and  9(f)  is  the 
phase  function  of  F(f)  (Ref  6:78). 

While  this  interpretation  is  admittedly  liberal,  a more 
formal  relationship  can  be  seen  by  considering  the  FOBW 
process  as  a zero-crossing  analysis  method.  It  can  be  shown 
(Refs  2 and  3)  that,  except  for  an  arbitrary  scale  factor. 
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Che  Fourier  coefficients  of  a function  whose  Fourier  transform 
is  band-limited  can  be  derived  from  the  zero-crossings  of  the 
function.  This,  in  turn,  suggests  a relationship  between  the 
mechanism  by  which  the  FOBW  method  operates  on  waveform  data 
and  the  Fourier  transform  of  the  signal. 
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IV.  Target  Classification  by  the  FOBW  Method 


Data  Base 

The  original  data  collection  process  was  accomplished 
under  government  contract  by  the  Environmental  Research 
Institute  of  Michigan  (ERIM)  of  Ann  Arbor,  Michigan.  Three 
targets  were  illuminated  as  they  passed  through  an  estab- 
lished range  gate,  and  the  return  spectra  were  recorded  at 
16  different  aspect  angles.  Target  velocities  ranged  from 
approximately  5 mph  to  approximately  15  mph,  and  the  aspect 
angles  varied  from  0 to  360  degrees  in  22.5  degree  incre- 
ments. The  ERIM  data  was  digitized  by  Maj  C.  V.  Stewart, 
an  Air  Force  Institute  of  Technology  doctoral  student,  in 
conjunction  with  his  own  dissertation  using  the  same  data. 
The  digitization  was  accomplished  by  sampling  the  analog 
data  at  2 KHz  and  recording  the  results  on  magnetic  tape. 
Both  the  analog  and  digitized  data  were  labeled  as  a series 
of  runs,  with  each  run  number  corresponding  to  the  particu- 
lar aspect  angle  at  which  the  original  data  was  collected. 

Both  inphase  and  quadrature  components  of  the  received 
video  signal  were  recorded  and  digitized.  Since  no  particu- 
lar advantage  could  be  seen  to  using  either  component,  it 
was  decided  arbitrarily  to  process  the  inphase  signal.  The 
digitized  data  was  divided  into  a number  of  samples,  where 
each  sample  consisted  of  1024  adjacent  points  in  the  inphase 
signal.  This  sample  length  was  selected  for  convenience  due 
to  the  manner  in  which  the  tapes  were  prepared  and  to  insure 
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that  the  resulting  binary  sequence  would  be  of  adequate 
length.  The  run  numbers  and  corresponding  aspect  angles 
are  tabulated  for  each  target  in  Appendices  A and  B. 

Preprocessing 

The  flow  chart  of  the  entire  pattern  recognition  algo- 
rithm is  shown  in  Fig.  4.  The  main  program  of  the  algorithm 
selects  the  desired  samples  of  the  digitized  data  and  all 
subsequent  processing  is  done  via  subroutine  calls.  The 
preprocessing  subroutines  are  enclosed  by  the  dashed  lines 
in  Fig.  4;  each  sample  is  time  averaged,  velocity  normalized, 
and  clipped  and  coded  to  a binary  sequence  which  is  then 
scanned  for  the  frequency  of  occurrence  of  delayed  digrams. 
Plot  subroutines  were  also  developed  which  display  the  un- 
averaged lnphase  signal  (I),  the  averaged  inphase  signal, 
the  quadrature  signal  (Q) , and  the  phase  (Arctan  Q/I)  of  the 
complex  signal.  It  was  stated  earlier  that  the  FOBW  method 
is  rather  simple  to  apply  in  practice,  and  preprocessing  the 
data  prior  to  implementing  the  delayed  digram  search  algo- 
rithm proved  to  be  more  difficult  than  implementing  the 
search  algorithm  itself. 

The  time  averaging  subroutine  was  used  to  remove  the 
ground  clutter  in  which  the  target  signature  was  immersed. 

The  ground  clutter  had  the  effect  of  forming  a DC  bias  which 
varied  considerably  among  samples.  Digital  filtering  may 
have  been  a more  effective  method  of  removing  the  bias,  but 
the  time  averaging  routine  appeared  to  work  sufficiently 
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well  and  was  easier  to  Implement 


It  was  observed  that 


averaging  the  inphase  signal  in  64  point  increments  not  only 
effectively  removed  the  DC  bias,  but  also  produced  a wave- 
form which  had  a high  degree  of  symmetry  about  the  zero  axis. 
Figures  5 and  6 illustrate  this  effect;  Fig.  5 shows  the 
asymmetry  of  an  unaveraged  sample  from  target  1,  and  Fig.  6 
depicts  the  same  sample  after  time  averaging  has  occurred. 

This  symmetry  feature  in  the  averaged  signal  was  quite  useful 
in  the  velocity  normalization  subroutine  which  was  called 
next . 

Intuitively,  it  would  be  desirable  for  each  of  the 
targets  to  have  the  same  uniform  velocity  before  any  identi- 
fication attempts  are  made.  Velocity  differentials  among 
targets  and  among  individual  samples  from  the  same  target 
would  produce  radically  different  zero-crossing  and  binary 
word  characteristics.  This,  in  turn,  would  make  the  feature 
selection  process  difficult,  if  not  impossible.  Since  it  is 
unrealistic  to  impose  a single  constant  velocity  criteria  in 
practical  applications,  a normalization  subroutine  was  de- 
veloped which  normalizes  all  target  velocities  to  approxi- 
mately 18  mph.  This  is  somewhat  higher  than  the  velocities 
observed  in  the  data  collection  process,  but  normalizing  to 
a higher  velocity  insures  that  all  normalizations  will  be 
in  the  same  "direction"  and  simplifies  the  algorithm. 

As  mentioned  earlier,  the  symmetry  feature  of  the  time 
averaged  inphase  signal  was  used  in  the  normalization  process. 
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Since  the  averaged  signals  were  symmetric  about  the  zero  axis, 
the  number  of  periods  in  each  sample  could  be  determined  by  a 
zero  crossing  count.  Since  the  radar  returns  are  available 
in  a discrete  analog  form,  one  could  normalize  a lower  veloc- 
ity (frequency)  sample  to  a higher  velocity  (frequency)  refer- 
ence by  inserting  points  in  the  unnormalized  waveform  at  the 

rate  T /T  , where  T is  the  number  of  periods  one  would  ob- 
r 8 r 

serve  in  the  reference  sample  and  Tg  is  the  number  of  periods 
present  in  the  unnormalized  sample.  Figure  7 shows  the  result 
of  normalizing  a waveform  upward  to  twice  its  frequency  and 
the  binary  sequences  which  would  arise  after  sampling  and 
coding . 

The  simplest  case  would  arise  if  the  ratio  T /T  is  some 

r s 

integer  I;  a normalized  waveform  could  be  obtained  by  retain- 
ing every  Ith  discrete  point  in  the  original  sample,  as  in 
Fig.  7.  Since  the  normalized  waveform  eventually  will  be 
infinitely  clipped,  the  magnitudes  selected  for  the  inserted 
points  are  unimportant  and  only  their  algebraic  signs  are 
significant . 

If  Tr/Ts  is  not  an  integer,  the  points  in  the  original 
sample  again  may  be  used  to  generate  a normalized  waveform; 
as  each  point  is  Inserted  into  the  unnormalized  sample,  it 
could  be  given  the  magnitude  and  sign  of  the  nearest  point 
already  present.  In  this  case,  there  is  a potential  for 
occassional  errors  when  the  inserted  points  are  placed  in 
the  vicinity  of  a zero  crossing  in  the  original  waveform. 


21 


Unnormalized  Signal 
Sampled  and 
Binarized  to 
11110000001111 


Normalized  Signal 
Sampled  and 
Binarized  to 
1100011 


Reference  Signal 
Sampled  and 
Binarized  to 
11000110001100 


Fig.  7.  Waveform  Normalization. 


These  errors  would  be  counted  only  as  end  effects  in  delayed 
digram  frequency  calculations,  however,  and  It  was  thought 
that  those  calculations  affected  by  errors  might  still  be 
suitable  approximations. 

The  clipping  and  coding  subroutines  follow  the  proce- 
dure described  in  Chapter  III.  At  this  point,  the  time 
averaged,  normalized  waveform  is  still  in  digitized  form 
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and  is  easily  converted  to  a binary  sequence  by  coding  all 
positive  points  as  a 1 and  coding  all  other  points  as  a 0. 

The  binary  sequence  which  results,  however,  has  somewhat 
fewer  points  than  vere  present  in  the  original  sample. 

This  is  due  to  the  normalization  process;  since  the  target 
velocities  are  being  normalized  upward  to  a velocity  greater 
than  that  of  any  of  the  original  samples,  the  ratio  ^T/^B  is 
always  greater  than  unity.  As  Tr/Tg  is  also  the  rate  or 
frequency  of  point  insertion,  one  would  exceed  the  original 
sample  length  before  1024  points  could  be  Inserted.  To  re- 
flect this  variation  in  sequence  length  among  samples,  it 
was  necessary  to  weight  each  calculated  frequency  of  occur- 
rence by  the  length  of  the  particular  sequence  from  which  it 
came . 

Prototype  Generation 

Ideally,  one  could  generate  prototypes  for  the  FOBW 
method  rather  easily  by  calculating  the  frequencies  of 
several  hundred  n-grams  or  delayed  digrams.  Ideally,  at 
least  a few  of  the  calculated  frequencies  of  occurrence  for 
the  members  of  a given  class  would  differ  substantially  from 
those  of  the  remaining  classes.  By  creating  the  appropriate 
number  of  two  class  problems,  one  could  then  separate  each 
class  by  the  magnitude  of  the  selected  frequencies.  Although 
prototypes  were  generated  for  each  target  in  this  fashion,  it 
was  somewhat  more  difficult  than  expected;  all  of  the  most 
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promising  features  eventually  showed  some  dependence  on 
target  aspect  angle. 


The  first  attempt  at  target  classification  was  to  sep- 
arate target  1 from  targets  2 and  3.  Targets  2 and  3 were 
very  nearly  identical  and  differed  only  by  a structural  modi- 
fication. Target  1,  on  the  other  hand,  was  the  only  repre- 
sentative of  its  class  and  was  particularly  unique  in  its 
means  of  locomotion.  It  was  also  decided  to  begin  the  feature 
extraction  process  with  an  arbitrarily  small  subset  of  the 
total  available  data  and  to  increase  the  size  of  the  training 
and  test  sets  as  promising  attributes  emerged.  This  made  for 
a reasonably  short  computer  turn-around  time  in  this  phase  of 
the  problem  and,  since  the  attributes  were  selected  heuristi- 
cally,  kept  the  initial  amount  of  data  to  a manageable  level. 

The  frequencies  of  occurrence  of  the  delayed  digrams 
1-m-O,  1-m-l,  and  0-m-0  were  calculated  for  the  first  five 
runs  of  each  target  as  m varied  from  1 to  50.  Sixty  n-gram 
frequencies  also  were  calculated  for  the  same  runs  as  n varied 
from  2 to  5.  A training  set  was  generated  by  setting  aside 
the  first  sample  of  each  run,  and  the  selected  features  were 
tested  against  the  remaining  samples.  The  feature  selection 
process  was  entirely  heuristic,  and  those  frequencies  which 
appeared  to  have  unique  threshold  magnitudes  for  each  class 
of  targets  were  selected  as  tentative  prototypes.  The  attri- 
butes we.«  tested  individually  on  their  ability  to  separate 
the  targets;  this  would  readily  identify  the  set  of  optimal 


24 


features  for  consideration  should  separability  be  improved 
by  the  use  of  multiple  attributes. 

The  initial  training  and  test  sets  were  augmented  in- 
crementally until  the  entire  collection  of  data  had  been 
processed.  As  before,  the  first  sample  of  each  additional 
run  was  placed  in  the  training  set  and  the  remaining  samples 
were  used  in  testing.  The  frequency  of  occurrence  thresholds 
were  adjusted  to  fit  each  new  training  set  and  were  then 
evaluated  in  the  cumulative  test  set.  As  the  size  of  the 
training  and  test  sets  increased,  the  following  observations 
were  made. 

1.  As  m increased  in  the  l-m-0,  1-m-l,  and  0-m-0 
delayed  digrams,  the  ability  of  the  corresponding  frequency 
of  occurrence  to  separate  the  targets  did  not  improve  over 
that  of  the  features  already  selected.  In  the  initial 
training  and  test  set,  where  m varied  from  1 to  50,  the 
longest  delayed  digram  attribute  selected  was  0-19-0  for  the 
separation  of  targets  2 and  3,  and  1-17-0  for  the  separation 
of  target  1 from  targets  2 and  3.  When  approximately  half 
of  the  total  available  data  had  been  examined,  the  process 
was  repeated  with  m varying  from  1 to  200  in  an  attempt  to 
Improve  separability.  None  of  the  new  attributes  generated, 
however,  could  be  identified  as  being  as  effective  as  those 
already  in  use. 

2.  The  n-gram  frequencies  of  occurrence  were  simi- 
lar for  all  targets  to  such  an  extreme  as  to  preclude  their 
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use  as  effective  attributes.  While  this  does  not  necessar- 
ily infer  that  larger  n-grams  would  also  constitute  poor 
features,  the  comparative  success  of  the  delayed  digram 
frequencies  prompted  a decision  to  disregard  the  n-gram 
frequencies  as  likely  attributes. 

3.  No  single  digram  frequency  was  particularly 
effective  when  used  alone.  Each  of  the  attributes  selected 
from  the  initial  training  set  eventually  displayed  an  aspect 
angle  dependence,  and  the  calculated  frequencies  of  occurrence 
for  a given  target  in  these  s ituat ions s tr ayed  into  the  range 
of  values  set  aside  for  at  least  one  other  target. 

The  heuristic  search  described  above  produced  six  delayed 
digram  frequencies  which,  when  used  with  the  decision  rule 
from  the  following  section,  best  separated  the  targets  in  the 
final  test  set.  The  frequencies  and  fo-3-0 

were  given  threshold  values  of  0.850,  0.241,  and  0.092, 
respectively,  and  used  to  separate  target  1 from  targets  2 
and  3;  target  1 generally  had  1-3-0  frequencies  less  than 
0.850,  1-5-0  frequencies  greater  than  0.241,  and  0-3-0  fre- 
quencies greater  than  0.092,  while  targets  2 and  3 did  not. 

In  a similar  fashion,  the  frequencies  f^  ^ and 

*1-14-1  were  8iven  threshold  values  of  0.042,  0.064,  and 
0.053,  respectively,  and  used  to  separate  target  2 from 
target  3;  target  2 tended  to  have  frequencies  below  the 
threshold  level  for  all  three  delayed  digrams,  while  those 
of  target  3 were  generally  higher.  In  selecting  these 
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particular  frequencies,  consideration  was  given  primarily 
to  their  effectiveness  as  a group  when  used  with  the  fol- 
lowing decision  function. 

Decision  Function 

As  mentioned  in  the  previous  section,  all  of  the 
selected  attributes  displayed  some  degree  of  aspect  angle 
dependence.  The  resulting  overlap  of  frequencies  of  occur- 
rence for  different  targets  precluded  the  use  of  a single 
attribute  and  corresponding  simple  decision  surface  for  each 
separation.  In  the  test  of  a given  sample,  however,  if 
multiple  features  are  used  whose  frequencies  do  not  overlap 
simultaneously,  one  could  assign  a vote  to  the  outcome  of 
each  individual  test  and  make  a class  assignment  by  majority 
rule  or  reject  the  sample  in  the  event  of  a tie.  This  de- 
cision logic  is  similar  to  that  employed  by  committee 
machines,  and  is  capable  of  implementing  quite  complex 
decision  surfaces  (Ref  5:96). 

The  final  test  set  was  subjected  to  the  form  of  decision 

rule  described  above;  each  sample  was  first  tested  against 

the  criteria  for  target  1 ^ess  than  0.850,  f^ 

greater  than  0.241,  and  3 0 8reater  than  0.092),  and,  if 

at  least  two  of  these  criteria  were  met,  classified  as 

target  1.  If  two  or  more  of  the  criteria  failed,  the  sample 

was  tested  against  the  criteria  for  target  2 C f ^ _ 3 _ less 

than  0.042,  f,  Q , less  than  0.064,  and  f,  ..  , less  than 
l-o-l  1-14-1 

0.053).  If  two  or  more  of  the  criteria  were  met,  the  sample 
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was  classified  as  target  2.  Those  samples  which  failed 
both  the  target  1 and  target  2 tests  were  classified  as 
target  3.  Since  three  features  were  evaluated  in  each 
test,  the  algorithm  always  made  a classification  decision. 
Several  attempts  were  made  to  incorporate  a reject  option 
with  the  use  of  an  even  number  of  attributes,  but  the  added 
features  caused  more  incorrect  assignments  to  be  made  than 
were  saved  by  rejecting. 

Performance 

The  overall  performance  of  the  FOBW  algorithm  is  re- 
flected in  the  confusion  matrices  shown  in  Figs.  8 and  9; 
Fig.  8 displays  the  training  and  test  set  matrices  for  a 
three  class  problem,  while  Fig.  9 depicts  the  matrices  with 
targets  2 and  3 combined  in  a single  class.  A more  detailed 
accounting  of  the  performance  is  presented  in  Appendices  A 
and  8,  which  contain  the  training  and  test  set  results  for 
each  run  and  aspect  angle  in  the  three  class  problem  and  by 
aspect  angle  alone  on  the  two  class  problem. 

As  can  be  seen  from  Fig.  8,  the  algorithm  achieved  an 
overall  success  rate  of  67.5%  in  the  three  class  problem 
and  had  a reject  rate  of  15.5%.  The  low  percentage  of  cor- 
rect classification  is  due  largely  to  the  inability  of  the 
algorithm  to  distinguish  between  targets  2 and  3.  It  was 
mentioned  earlier  that  targets  2 and  3 differed  only  by  a 
structural  modification,  however,  and  one  might  suspect  that 
they  would  be  the  most  difficult  to  separate.  Since  the 
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Assigned 
12  3 

1 26  0 3 

2 2 5 4 

3 4 4 14 

Training  Set 


Target  1:  26  correct  for  89. 7% 

Target  2:  5 correct  for  45.4% 

Target  3:  14  correct  for  63.6% 

Overall:  45  correct  for  72.6% 


Assigned 
12  3 

1 337  3 30 

2 7 98  55 

3 61  158  218 

Test  Set 


Target  1:  337  correct  for  91.1% 

Target  2:  98  correct  for  61.2% 

Target  3:  218  correct  for  49.9% 

Overall:  653  correct  for  67.5 % 

177  rejects  for  15.5% 


Fig.  8.  Training  and  Test  Set  Confusion  Matrices  for  the 
Three  Class  Problem. 
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Test  Set 


Target  1:  337  correct  for  91.1% 

Target  2/3:  529  correct  for  88.6%. 

Overall:  866  correct  for  89.6% 

177  rejects  for  15.5% 


similarities  in  these  two  targets  are  so  pronounced,  it 
would  not  be  unreasonable  to  combine  them  into  a single 
class  and  examine  the  performance  of  the  FOBW  algorithm  in 
a two  class  problem.  From  Fig.  9,  this  results  in  an  over- 
all success  rate  of  89.6%. 

The  rejects  shown  in  the  confusion  matrices  arose  from 
an  effort  to  discard  any  sample  which  was  recorded  when  the 
target  was  not  established  in  the  range  gate.  The  majority 
of  the  runs  from  target  1 contained  target  returns  which  were 
sandwiched  between  empty  range  gate  returns.  The  same  was 
true  but  to  a lesser  extent  for  targets  2 and  3.  Samples 
from  the  empty  range  gate  typically  had  noise-like  spectra 
as  shown  in  Fig.  10,  and  could  not  be  used  in  classification 
attempts . 

Several  plots  of  the  inphase  signal  were  made  to  deter- 
mine the  location  and  duration  of  the  target  range  gate 
samples  in  the  first  few  runs  from  target  1,  and  it  was  ob- 
served that  samples  with  the  target  in  the  range  gate  typical- 
ly had  frequencies  of  occurrence  of  0.350  or  greater  for  the 
delayed  digram  1-1-0.  This  was  also  evident  in  the  data 
from  targets  2 and  3,  and  it  was  anticipated  that  rejecting 
those  samples  below  this  1-1-0  threshold  would  be  a more  effi- 
cient method  of  identifying  target  range  gate  data  than  plot- 
ting would  be. 

It  became  apparent,  however,  that  this  reject  mechanism 
occassionally  rejected  samples  corresponding  to  target  range 
gate  data.  This  was  more  evident  in  some  runs  than  in  others. 
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Sample 


and  accounts  for  the  small  number  of  samples  In  many  of 


these  runs.  The  number  of  rejects  In  each  run  was  calcu- 
lated by  recording  those  rejects  which  occurred  only  when 
the  target  was  established  in  the  range  gate  as  determined 
by  the  1-1-0  threshold  criteria. 

One  may  also  note  from  the  appendices  that  those  runs 
corresponding  to  aspect  angles  of  90  or  270  degrees  were 
not  used.  The  signatures  of  returns  observed  at  odd  multi- 
a of  90  degrees  are  void  of  any  Doppler  effect;  the  spec- 
tral sidebands  tend  to  collapse  upon  one  another  and  thereby 
limit  the  information  content  of  the  signature.  It  was 
thought  beforehand  that  data  from  these  returns  might  con- 
sistently fail  in  identification  attempts,  and  this  was 
verified  in  separate  trials. 


I 

I 


™ 1 

r . Conclusions  and  Recommendations 

The  FOBW  method  has  been  examined  as  a time  domain 
recognition  algorithm  for  use  in  classifying  radar  signa- 
tures.  From  an  evaluation  of  the  performance  of  the  method, 
the  following  conclusions  were  reached. 

1.  The  FOBW  method  does  have  merit  as  a target 
classification  algorithm.  Considering  the  simplistic  ap- 

’ proaches  to  the  time  averaging  and  velocity  normalization 
programs,  the  principle  flaw  in  the  performance  of  the  method 
is  the  15%  reject  rate.  This  might  be  improved,  however,  by 
the  use  of  more  effective  averaging  and  normalization  rou- 
tines. 

2.  The  FOBW  method  may  be  aspect  angle  invariant; 
although  individual  frequencies  showed  an  angle  dependence, 
the  use  of  multiple  frequencies  did  not  show  any  correlation 
between  errors  or  rejects  and  a particular  aspect  angle. 

3.  The  higher  frequency  spectral  components  appear 
to  contribute  to  target  classification  by  this  method.  A 
short  l-m-0  digram  provides  an  indication  of  the  distance 
between  the  zero-crossings  of  a waveform,  and  the  frequency 
of  occurrence  of  two  such  short  digrams  constituted  two 
thirds  of  the  decision  rule  in  the  two  class  problem. 

In  light  of  the  first  conclusion  reached  above,  the 
following  recommendations  are  made. 

1.  The  FOBW  method  should  be  applied  to  other 
data  bases  to  determine  its  general  applicability  in  target 
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classification  problems. 

2.  The  time  averaging  and  velocity  normalization 
programs  should  be  altered  to  determine  if  the  performance 
can  be  improved.  Digital  filtering  may  be  a more  effective 
method  of  removing  the  ground  clutter,  and  it  can  be  shown 
that  target  velocity  is  related  to  the  phase  of  a radar 
return.  These  or  other  variations  may  result  in  a tighter 
clustering  of  digram  frequencies  and  may  enhance  the  feature 
selection  and  decision  processes. 

3.  Variations  in  sample  lengths  should  be  examined. 
Doubling  the  sample  length,  while  decreasing  the  total  number 


of  samples,  may 

provide  better  estimates 

for  the 

f requen 

cies 

of  occurrence. 

Conversely , 

reducing  the 

sample 

length , 

if 

the  frequency  of 

occurrence 

estimates  are 

still 

acceptab 

le. 

would  provide  a 

larger  test 

set . 
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Appendix  A 


! 


i 


| 


Training  and  Test  Set  Results  for 
the  Three  Class  Problem 

Appendix  A contains  the  training  and  test  set  results 
for  the  three  class  recognition  problem.  Entries  are  tabu- 
lated by  target  number,  run  number,  and  aspect  angle,  and, 
in  the  event  of  an  incorrect  classification,  the  target 
selected  by  the  algorithm  is  given.  For  the  test  set  re- 
sults, the  number  of  rejects  in  each  run  is  also  tabulated. 
Omitted  run  numbers  correspond  to  either  blank  runs  or 
data  collected  at  90  or  270  degrees. 


Target  1 Tralnin 


Classified  as 


Run 

No. 

Aspect 

Angle 

Targe 

366 

337.5 

X 

367 

157.5 

X 

368 

337.5 

X 

369 

157.5 

X 

370 

315 

X 

Target  2 


Target  3 


Run 

No. 


I 


382 

384 

385 

386 

387 

392 

393 

394 

395 

396 

397 


Target  2 Training  Set 

Classified  as 

Aspect 

Angle 

Target  1 Target  2 

Target  3 

157.5 

X 

135 

X 

315 

X 

67.5 

X 

292.5 

X 

67.5 

X 

247.5 

X 

045 

X 

22.5 

X 

22.5 

X 

202.5 

X 

2 5 

4 

Aspect 

Angle 


202.5 

202.5 


247.5 

247.5 


112.5 


337.5 

157.5 


22.5 


202.5 


67.5 

247.5 


Target  3 Training  Set 


Target  1 


Classified  as 


Target  2 


Target  3 


Target  1 Test  Set 


350  202 


423  337 
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Appendix  B 


Training  and  Test  Set  Results  for 
the  Two  Class  Problem 

Appendix  B contains  the  training  and  test  set  results 
for  the  two  class  problem  (target  1 vs.  targets  2 and  3). 
The  results  for  each  target  are  presented  separately  and 
are  tabulated  by  aspect  angle  alone. 
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, u 


- >1 


)1 


1 


j 


Aspect 
Angle 

360 

22.5 
45 

67.5 

112.5 
135 

157.5 
180 

202.5 
225 

247.5 

292.5 
315 

337.5 


Target  1^  Training  Set 


No.  of  Samples  Classified  as 


Target  1 
1 
2 
1 
1 
1 
3 

3 
1 
1 

4 
1 
1 
3 

_3 

26 
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Target  2/3 


1 

1 


w 


7 


Aspect 

Angle 

360 

22.5 
045 

67.5 

112.5 
135 

157.5 
180 

202.5 
225 

247.5 

292.5 
315 

337.5 


Target  Training  Set 

No.  of  Samples  Classified  as 
Target  1 Target  2/3 


2 

1 

1 

1 

1 


1 

1 
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Aspect 
Angle 


22.5 


67.5 


112.5 


157.5 


202.5 


Target  3 Training  Set 


No.  of  Samples  Classified 


Target  1 


Target  2/3 


247.5 


292.5 


337.5 


P 


Target  1^  Test  Set 


No.  of  Samples 
Classified  as 


Aspect 

Angle 

No . of 
Samples 

Target  1 

Target  2/3 

No . of 
Re j ects 

360 

26 

24 

2 

1 

22.5 

43 

39 

4 

1 

045 

28 

26 

2 

6 

67.5 

14 

14 

- 

3 

112.5 

10 

7 

3 

1 

135 

34 

33 

1 

11 

157.5 

34 

30 

4 

8 

180 

24 

21 

3 

0 

202.5 

20 

19 

1 

2 

225 

40 

40 

- 

12 

247.5 

5 

4 

1 

4 

292.5 

8 

5 

3 

3 

315 

39 

33 

6 

9 

337.5 

45 

42 

3 

21 

370 

337 

33 

82 

52 


No.  of  Samples 
Cl a ssified  as 


Aspect 

Angle 

No.  of 
Samples 

Target  1 

Target  2/3 

No . of 
Rel ects 

360 

72 

6 

66 

10 

22.5 

31 

1 

30 

5 

045 

54 

4 

50 

16 

67.5 

11 

6 

5 

5 

112.5 

12 

6 

6 

5 

135 

24 

1 

23 

0 

157.5 

25 

3 

22 

0 

180 

44 

- 

44 

6 

202.5 

66 

11 

55 

6 

225 

21 

3 

18 

3 

247.5 

29 

13 

16 

20 

292.5 

- 

- 

- 

- 

315 

19 

- 

19 

7 

337.5 

29 

7 

22 

2 

437 

61 

376 

85 

54 


Appendix  C 


FOBW  FORTRAN  Algorithm 

Appendix  C contains  the  FOBW  FORTRAN  algorithm  used 
to  classify  the  targets.  The  algorithm  is  commented  where 
appropriate,  and  should  be  self  explanatory. 
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Appendix  D 

Heuristic  Search  Procedure 

Appendix  D contains  an  example  of  the  heuristic  search 
procedure  used  to  determine  the  delayed  digram  frequency  of 
occurrence  attributes  and  their  threshold  levels.  It  may 
be  recalled  that  the  first  five  runs  of  each  target  were 
set  aside  as  an  initial  training  set  and  the  frequencies 
of  occurrence  of  the  delayed  digrams  l-m-0,  1-m-l,  and 
0-m-0  were  calculated  as  m varied  from  1 to  50.  An  abbre- 
viated version  of  the  procedure  followed  in  analyzing  the 
delayed  digram  frequencies  is  shown  below;  the  example 
depicts  the  selection  of  promising  l-m-0  attributes  for  the 
two  class  problem  (target  1 vs.  targets  2 and  3)  as  m varies 
from  1 to  10. 

Tables  1 and  2 show  the  l-m-0  delayed  digram  frequen- 
cies and  their  means  for  all  three  targets.  One  method  of 
selecting  attributes  would  be  to  compare  the  frequency  means 
to  determine  if  any  obvious  dissimilarities  exist  among  the 
targets.  If  desired,  one  could  also  calculate  the  variances 
for  an  indication  of  how  well  the  individual  frequencies 
appear  to  cluster  about  each  mean.  The  comparison  of  the 
means  in  Table  2 is  perhaps  not  as  striking  as  one  would 
desire,  and  a subjective  decision  as  to  how  near  the  means 
may  approach  one  another  before  a particular  frequency  of 
occurrence  is  discarded  as  a likely  attribute  is  required. 
Since  targets  2 and  3 were  considered  as  a single  class. 
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the  frequency  of  occurrence  means  from  both  targets  would 
be  considered  simultaneously,  also. 

The  heuristic  selection  of  attributes  Is  made  somewhat 
easier  by  a graphic  display  of  the  frequency  means.  Figure 
11  contains  a plot  of  the  frequency  means  for  each  target 
vs.  the  sampling  interval.  From  Fig.  11,  one  might  subjec- 
tively select  the  frequencies  of  occurrence  of  the  delayed 
digrams  1-3-0,  1-5-0,  1-6-0,  1-8-0,  and  1-10-0  as  those 
attributes  which  display  the  greatest  difference  in  magni- 
tudes between  the  frequencies  of  target  1 and  the  frequen- 
cies nearest  in  magnitude  of  targets  2 or  3.  It  is  also 
possible  to  observe  the  heuristic,  nonexhaus t ive  nature  of 
such  a decision;  with  few  relatively  large  differences  in 
the  means,  tentative  proto-type  selection  is  largely  a 
matter  of  judgement. 

Having  selected  the  attributes,  the  threshold  levels 
could  be  established  so  as  to  include  as  many  of  the  target 
training  samples  as  is  possible;  for  the  1-3-0  frequency, 
for  example,  a threshold  level  of  0.860  would  distinguish 
between  targets  2/3  and  target  1 in  all  but  one  instance 
(f^_3_0  for  sample  3,  target  3,  is  0.791).  By  classifying 
the  sample  as  target  1 if  f^  ^ q i®  below  0.860  and  classi- 
fying it  as  target  2 or  3 if  it  is  greater,  one  could  suc- 
cessfully classify  the  sample  in  14  of  15  attempts. 

The  remainder  of  the  attributes  could  be  selected  in 
a similar  fashion.  As  mentioned  earlier,  as  the  training 
and  test  sets  grow  larger,  one  would  simultaneously  discard 
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attributes  which  appear  less  promising  when  applied  to  a 
larger  number  of  samples  and  adjust  the  threshold  levels 
of  the  remaining  attributes  to  optimize  their  performance. 
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